Skip to content

[RNE Rewrite] feat: Add OCR implementation - #1322

Draft
benITo47 wants to merge 21 commits into
rne-rewritefrom
ocr_dynamic
Draft

[RNE Rewrite] feat: Add OCR implementation #1322
benITo47 wants to merge 21 commits into
rne-rewritefrom
ocr_dynamic

Conversation

@benITo47

Copy link
Copy Markdown
Contributor

Description

Adds a unified, OCR and document understanding pipeline to react-native-executorch. Two OCRs - EasyOCR and PP-OCRv6, plus a higher-level document pipeline that orchestrates orientation correction, UVDoc dewarp, PP-DocLayoutV3 region layout, SLANet table-structure recognition, and reading-order assembly into HTML.

Introduces a breaking change?

  • Yes
  • No

Type of change

  • Bug fix (change which fixes an issue)
  • New feature (change which adds functionality)
  • Documentation update (improves or adds clarity to existing documentation)
  • Other (chores, tests, code style improvements etc.)

Tested on

  • iOS
  • Android

Testing instructions

Try OCRing on different images, use different OCR options. See if anything is ill behaved.

Screenshots

Related issues

Checklist

  • I have performed a self-review of my code
  • I have commented my code, particularly in hard-to-understand areas
  • I have updated the documentation accordingly
  • My changes generate no new warnings

Additional notes

benITo47 added 12 commits July 20, 2026 19:17
…OcrOptions.tables)

Tables were the odd one out — orientation/dewarp had per-run toggles, tables
fired unconditionally when a table region was found. Add RunOcrOptions.tables
(default true when the model supports them; a per-run false skips the
recognizer), gate the table pass on it, and surface a Table-structure sub-toggle
in the demo alongside orientation/dewarp.
…ailing | _ glyphs)

Flat-gray strip padding left a content→pad seam EasyOCR's CRNN read as a
trailing glyph. Add OcrModelOptions.recognizerPadMode ('constant' default,
'cornerMean' = background-matched), thread it through the recognizer, default
EasyOCR to cornerMean. Not a decode bug — the argmax was predicting the seam.
…f composed strip

Composing a vertical stack into one horizontal strip fed the recognizer a
non-word column with its horizontal-word context, making it hallucinate words
(and the vertical-digit phantoms). Recognize each glyph on its own and join
top-to-bottom — drops the word bias, gives per-glyph confidence. Cost: one
recognizer pass per glyph (fine for the opt-in vertical path).
Vertical reads skip the horizontal drop-score gate because they run
lower-confidence. Expose RunOcrOptions.verticalMinConfidence so callers can
raise it to filter the low-confidence hallucinations the vertical path emits.
@benITo47 benITo47 changed the title Ocr dynamic [RNE Rewrite] feat: Add OCR implementation Jul 20, 2026
Hardens the pipeline — encodeConstraint variant handling, snapUpDim lattice clamp, span-aware table fill, tagged shape-constraint union, dead compose-mode removal, JSI length guards and demo polish. Also pins the document pipeline to the now-published 0.9.0 model tag.
Drops the PR's get_dynamic_dims_/get_enum_shapes_ companion machinery in
cpp/core (model.cpp/h, tensor_helpers.*) and src/core/model.ts in favour of
the get_model_schema contract that landed in #1327/#1346.
…chema contract

Replaces the removed get_dynamic_dims_/get_enum_shapes_ companion API with
validateSpec over model.schema:

- detect/recognize input sizes are now per-dimension ConcreteDim domains, so
  one snap routine covers constant/range/enum and the CoreML enumerated grid
  needs no cross-dimension shape set.
- the recognizer's width-to-CTC-timestep relation is read from the linear
  runtime constraint the model declares instead of inferred as a width/timestep
  ratio, which was wrong for EasyOCR's CRNN (512 = 4*127 + 4, not a multiple).
- TextBoxExtractor declares the detect output layout it decodes, so the allowed
  spec covers the outputs instead of wildcarding them.
- documentModels validates its four static methods in one spec; repeated
  symbols now enforce the vocab/hidden/feature agreements the manual checks did.
- useOcr resolves its nested models through the single-pass resource download.
…HF repos

The four OCR repos now follow the MODEL_SPEC layout the rest of the ecosystem
uses — per-backend directories, a root config.json, and
<model>[_<size>]_<backend>_<precision>.pte file names — published under
v0.10.0, so they share NEXT_VERSION_TAG with the other rewrite models instead
of the unprefixed 0.9.0 tag they had.

Precision is now part of the path, so each preset spells out what it actually
ships rather than hiding it behind a bare backend name.
PP-DocLayoutV3's RT-DETR head is set prediction, so the preset runs it with
NMS off — but it still emits nested duplicates, e.g. a region for one line
inside the region for the paragraph that contains it. Every region is OCR'd on
its own crop, so those duplicates transcribed the same text twice: a page with
"Lekcja / Indywidualna" came back as three blocks, one per nesting.

IoU suppression cannot fix this (a small box inside a large one has low IoU),
so the regions are now merged on containment, as PaddleX's
`layout_merge_bboxes_mode: 'large'` does: largest first, drop any region a kept
one already covers by >=80% of its own area. Visual regions never suppress, so
a caption inside a figure is still read. It also stops recognizing those crops
twice.
…very OCR backend

The OCR and Model Inspector screens never read the safe-area insets the other
eight screens already pad by, so "Run OCR" sat under the navigation bar, and
the main menu was a fixed View whose tenth button was clipped by it. The menu
scrolls now, still centered while it fits.

The OCR screen also filtered its model list by platform instead of disabling
what the platform cannot run, so CoreML was invisible on Android. All six
variants are listed now, with CoreML disabled off iOS and Vulkan disabled on
it, matching every other screen.
@msluszniak msluszniak linked an issue Aug 12, 2026 that may be closed by this pull request
Brings the OCR pipeline in line with the conventions that landed on
rne-rewrite while this branch was open.

- The three OCR-specific native ops were called straight off
  `rnexecutorchJsi.cv` from inside the task files. add-native-extension
  requires task-specific ops to be wrapped in
  `src/extensions/<domain>/utils/<name>.ts` and re-exported from the domain
  index, the way vadUtils/supertonicUtils/privacyFilterUtils are, so power
  users can reach them: new `cv/utils/ocrUtils.ts` with typed, documented
  `extractCraftTextBoxes` / `extractDbnetTextBoxes` / `ctcGreedyDecode`.
- Five bare `throw new Error` (geometry, detectors, documentModels, quad ops)
  now carry codes: INVALID_ARGUMENT for a caller's bad input, INVALID_STATE
  for calling table recognition on models loaded without a table config.
  Documented with `@throws` on the entry points that can raise them.
- `opts` parameters renamed to `options` (#1358): `makeCraftExtractBoxes`,
  `makeDbnetExtractBoxes`, `rectifyQuad`.
- `OcrModel.ocrOpts` -> `modelOpts`, matching every other task's model type.
- Tensor locals carry the `t` prefix the pipeline skill asks for.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

feature PRs that implement a new feature refactoring

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[RNE Rewrite] CV - add OCR pipeline implementation

2 participants